Movement and Gesture in Intelligent Interactive Music Systems

This paper introduces a research area on music and interaction by discussing issues on the integration of movement and music languages. As concerns movement, a special focus is given to high-level and qualitative analysis of human movement, starting from previous studies on movement from this viewpoint, such as Laban’s Theory of Effort (Laban and Lawrence 1947), and from KANSEI Information Processing (Camurri 1997; Hashimoto 1997). Typical application scenarios include interactive music, theatre, art and museum installations possibly including robots on stage. Other main research issues include real-time analysis and synthesis of expressivity in music and in movement signals (e.g., dance), multimodal interaction and intelligent interfaces where sound, music, as well as visual and gesture languages, mobile (robotic) scenographies are the components of an integrated multi-level communication. We present this research area and some recent developments by introducing our EyesWeb project and some music projects in which these issues have been faced. The EyesWeb project goals are twofold: from one hand, it aims at exploring and developing models of interaction by extending music language toward gesture and visual languages, with a particular focus on analysing expressive content in gesture and movement, and generating expressive outputs. For example, in EyesWeb we aim at developing methods able to distinguish the different expressive content from two instances of the same movement pattern, e.g., two performances of the same dance fragment. From the other hand, EyesWeb includes the development of an open, visual software for the real-time analysis of fullbody movement and gesture of one or more humans and its use in the scenarios described above. Our research is also inspired by the well-known languages for choreography Laban Theory of Effort (Laban and Lawrence 1947) and Eschol-Wachman and from research in KANSEI Information processing (Hashimoto 1997). We are successfully employing our research in EyesWeb in various music, interactive art and museal applications, shortly surveyed in the final part of the paper.